As computers become more mobile and ubiquitous, people increasingly expect always-available computing, either with devices that they carry on their bodies, or using devices embedded in the environment. There is an increasing need for interaction modalities that go beyond the keyboard and mouse, and furthermore, that do not require mediated interaction with specialized devices such as styluses.
Researchers have addressed this need through a variety of input channels. Speech recognition enables hands-free interaction for a variety of desktop and mobile applications. Similarly, computer vision enables machines to recognize faces, track movement, recognize gestures, and reconstruct three-dimensional (3d) scenes. Various techniques, most notably capacitive sensing, have been used to instrument surfaces such a tables, walls, and mobile devices in order to provide touch sensing. In addition, specialized depth cameras that allow users to interact with their computers using whole-body gestures have recently become commercially available to consumers (e.g., MICROSOFT™ KINECT™).
Speech input comes at a relatively low cost of instrumentation, but is limited in input bandwidth and may not be appropriate in many scenarios. Vision- and touch-based technologies offer an array of subtle, natural interaction techniques, but are limited in the potential scale of deployment due to their associated installation burden and cost. Consequently, there will likely not be homes or workplaces that allow truly ubiquitous input in the near future using these modalities.
Other researchers, realizing these limitations, have explored sensors that exploit characteristics of the human body itself to turn it into an inherently portable interaction device. Some have used bio-acoustic sensors to determine the location of taps on the body, and thereby turn it into a touchscreen. Others use electrical recordings of forearm muscles to sense muscle activity and infer finger gestures. However, these on-body input systems are to date limited to a small number of discrete inputs, and do not offer the large-scale interaction that is provided by touch-sensitive surfaces.
Touch sensing and computer vision have made human-computer interaction possible in environments where keyboards, mice, or other handheld implements are not available or desirable. However, the high cost of instrumenting environments limits the ubiquity of these technologies, particularly in home scenarios where cost constraints dominate installation decisions.